Natural Gradient for Combined Loss Using Wavelets
نویسندگان
چکیده
Natural gradients have been widely used in the optimization of loss functionals over probability space, with important examples such as Fisher–Rao gradient descent for Kullback–Leibler divergence, Wasserstein transport-related functionals, and Mahalanobis quadratic functionals. This note considers situation which is a convex linear combination these examples. We propose new natural algorithm by utilizing compactly supported wavelets to diagonalize approximately Hessian combined loss. Numerical results are included demonstrate efficiency proposed algorithm.
منابع مشابه
Monitoring Nonlinear Profiles Using Wavelets
In many manufacturing processes, the quality of a product is characterized by a non-linear relationship between a dependent variable and one or more independent variables. Using nonlinear regression for monitoring nonlinear profiles have been proposed in the literature of profile monitoring which is faced with two problems 1) the distribution of regression coefficients in small samples is unkno...
متن کاملGradient-based optimization of mother wavelets
We present a general framework for the design of a mother wavelet best adapted to a specific signal or to a class of signals. The filter’s coefficients are obtained via optimization of a smooth objective function. We develop an unconstrained gradient-based optimization algorithm for a discrete wavelet transform. The algorithm is extended to the joint optimization of the mother wavelet and of th...
متن کاملWavelets and Natural Image Statistics
It is well-known that wavelets provide a transformation of image data that has excellent properties with respect to image compression. The reasons for the compression ability of wavelets however have not been fully understood. In this paper we show that there is an interesting connection between wavelets and statistical properties of real-world images. This might lead to new theoretical and pra...
متن کاملWavelets and curvelets for image deconvolution: a combined approach
We propose in this paper a new deconvolution approach, which uses both the wavelet transform and the curvelet transform in order to bene2t from the advantages of each. We illustrate the results with simulations. ? 2003 Elsevier B.V. All rights reserved.
متن کاملEnergetic Natural Gradient Descent
In this appendix we show that 1 2 ∆ F (θ)∆ is a second order Taylor approximation of D KL (p(θ)p(θ + ∆)). First, let g q (θ) :=D KL (qp(θ)) = ω∈Ω q(ω) ln q(ω) p(ω|θ). We begin by deriving equations for the Jacobian and Hessian of g q at θ: ∂g q (θ) ∂θ = ω∈Ω q(ω) p(ω|θ) q(ω) ∂ ∂θ q(ω) p(ω|θ) = ω∈Ω q(ω) p(ω|θ) q(ω) −q(ω) ∂p(ω|θ) ∂θ p(ω|θ) 2 = ω∈Ω − q(ω) p(ω|θ) ∂p(ω|θ) ∂θ , (4) and so: ∂ 2 g q (θ)...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Journal of Scientific Computing
سال: 2021
ISSN: ['1573-7691', '0885-7474']
DOI: https://doi.org/10.1007/s10915-020-01367-x